Texture replacement in video sequences and images

ABSTRACT

Systems and methods for reducing bit rates by replacing original texture in a video sequence with synthesized texture. Reducing the bit rate of the video sequence begins by identifying and removing selected texture from frames in a video sequence. The removed texture is analyzed to generate texture parameters. New texture is synthesized using the texture, parameters in combination with a set of constraints. Then, the newly synthesized texture is mapped back into the frames of the video sequence from which the original texture was removed. The resulting frames are then encoded. The bit rate of the video sequence with the synthesized texture is less than the bit rate of the video sequence with the original texture. Also, the ability of a decoder to decode the new video sequence is not compromised because no assumptions are made about the texture synthesis capabilities of the decoder.

This application claims the benefit of U.S. Provisional PatentApplication Ser. No. 60/328,627, entitled “A Texture Replacement Methodat the Encoder for Bit Rate Reduction of Compressed Video,” filed Oct.11, 2001, which is incorporated herein by reference.

RELATED APPLICATIONS

This application is related to commonly assigned U.S. patent applicationSer. No. ______ entitled “System And Method For Encoding And DecodingUsing Texture Replacement,” filed ______ by Adriana Dumitras and BarinGeoffry Haskell and claiming priority to U.S. Provisional PatentApplication Ser. No. 60/360,027, filed Feb. 12, 2002. Thisabove-identified application is incorporated by reference herewith.

BACKGROUND OF THE INVENTION

1. The Field of the Invention

The present invention relates to systems and methods for reducing a bitrate of a video sequence. More particularly, the present inventionrelates to systems and methods for reducing a bit rate of a videosequence by replacing original texture of the video sequence withsynthesized texture at the encoder.

2. Background and Relevant Art

One of the goals of transmitting video sequences over computer networksis to have a relatively low bit rate while still maintaining a highquality video at the decoder. As technology improves and becomes moreaccessible, more users are leaving the realm of 56K modems and moving toDigital Subscriber Lines (DSL), including VDSL and ADSL, which support ahigher bit rate than 56K modems. VDSL, for example, supports bit ratesup to 28 Mbits/second, but the transmission distance is limited. Themaximum transmission distance for a 13 Mbits/second bit rate is 1.5 kmusing VDSL. ADSL, on the other hand, can support longer distances usingexisting loops while providing a bit rate of approximately 500kbits/second.

Video standards, such as MPEG-2, MPEG-4, and ITU H.263, can achieve bitrates of 3 to 9 Mbits/second, 64 kbits to 38.4 Mbits/second, and 8 kbitsto 1.5 Mbits/second, respectively. Even though video sequences with bitrates of hundreds of kbits/second can be achieved using these standards,the visual quality of these video sequences is unacceptably low,especially when the content of the video sequences is complex.

Solutions to this problem use model-based analysis-synthesis compressionmethods. Model-based analysis-synthesis compression methods perform bothanalysis and synthesis at the encoder to modify parameters in order tominimize the error between the synthesized model and the original. Theresulting parameters are transmitted to the decoder, which is requiredto synthesize the model again for the purpose of reconstructing thevideo sequence.

Much of the model-based analysis-synthesis compression methods havefocused on modeling human head-and-shoulders objects while fewerattempts have modeled background objects. Focusing on humanhead-and-shoulder objects often occurs because in many applications,such as videoconferencing applications, the background is very simple.However, background modeling may also achieve a significant reduction ofthe bit rate as the bit rate of I (intra) frames is often dependent onthe texture content of each picture. To a lesser extent, the bit rate ofB (bi-directionally predicted) frames and P (predicted) frames is alsoaffected by texture content as moving objects uncover additionalbackground objects.

One proposal for reducing the bit rate is to use sprite methods on thebackground objects. Sprites are panoramic pictures constructed using allof the background pixels that are visible over a set of video frames.Instead of coding each frame, the sprite is compressed and transmitted.The background image can be reconstructed using the sprite andassociated camera motion parameters. Sprite methods require exact objectsegmentation at the encoder, which is often a difficult task for complexvideo sequences. In addition, the motion or shape parameters that aretransmitted with the sprite consume some of the available bit rate.These limitations may be addressed by filtering the textured areas.Unfortunately, different filters must be designed for various textures.

Texture replacement has also been proposed as a method of backgroundmodeling. In one example, the original texture is replaced with anothertexture that is selected from a set of textures. However, this requiresthat the set of replacement textures be stored at the encoder. Inanother example, the texture of selected regions is replaced at theencoder with pixel values that represent an “illegal” color in the YUVcolor space. At the decoder, the processed regions are recovered usingchroma keying. There is an explicit assumption that texture synthesis,using texture parameters sent from the encoder, followed by mapping ofthe synthesized texture onto the decoded video sequences, is performedat the decoder. This method therefore assumes that the reconstruction isperformed at the decoder using a method that is dependent on thedecoder's processing capabilities. The drawbacks of these approaches arethat the processing capabilities of the decoder are assumed and that thecomputational costs of the decoding stage are increased.

BRIEF SUMMARY OF THE INVENTION

These and other limitations of the prior art are overcome by the presentinvention which relates to systems and methods for reducing the bit rateof a video sequence through texture replacement at the encoder. Thecapabilities of the decoder are not assumed and the decoder is notrequired to perform texture analysis and synthesis. The synthesizedtexture that replaces the original texture has similar perceptualcharacteristics to the original texture. Thus, the video sequence withthe synthesized texture is visually similar to the original videosequence. The bit rate of the video sequence with synthesized texturesis reduced because the synthesized textures that have replaced theoriginal textures can be coded more effectively.

Texture replacement, in accordance with the present invention, occurs atthe encoder and is therefore independent of the capabilities of thedecoder. Texture replacement begins by selecting and removing texturefrom some or all of the original frames in the video sequence. Theremoved texture is analyzed to obtain texture parameters. Then, newtexture is synthesized using the texture parameters in combination witha set of qualitative constraints. The synthesized texture can becompressed more effectively than the original texture and is alsosimilar to, yet distinguishable from, the original texture. After thenew texture is synthesized, the synthesized texture is inserted backinto the original frames and the video sequence that includes thesynthesized texture is encoded. Advantageously, the bit rate of thecompressed video sequence with the synthesized texture is lower than thebit rate of the compressed video sequence with the original texture.

Additional features and advantages of the invention will be set forth inthe description which follows, and in part will be obvious from thedescription, or may be learned by the practice of the invention. Thefeatures and advantages of the invention may be realized and obtained bymeans of the instruments and combinations particularly pointed out inthe appended claims. These and other features of the present inventionwill become more fully apparent from the following description andappended claims, or may be learned by the practice of the invention asset forth hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the above-recited and otheradvantages and features of the invention can be obtained, a moreparticular description of the invention briefly described above will berendered by reference to specific embodiments thereof which areillustrated in the appended drawings. Understanding that these drawingsdepict only typical embodiments of the invention and are not thereforeto be considered to be limiting of its scope, the invention will bedescribed and explained with additional specificity and detail throughthe use of the accompanying drawings in which:

FIG. 1 is a flow diagram illustrating an exemplary method for reducing abit rate of a video sequence by replacing original texture withsynthesized texture;

FIG. 2 illustrates a frame that includes various candidate textureswhich can be replaced with synthesized textures;

FIG. 3 is a flow diagram of a method for removing original texture fromvarious frames of a video sequence; and

FIG. 4 illustrates a recursive transform used in image decomposition toobtain texture parameters.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention relates to systems and methods for texturereplacement in video sequences and to reducing bit rates of videosequences through texture replacement. The bit rate of a compressedvideo sequence is reduced because the synthesized texture can be moreeffectively compressed than the original texture. Because the texturereplacement occurs as part of the encoding process, the computationalcosts of the decoder are not increased and the processing/synthesizingcapabilities of the decoder are not assumed.

FIG. 1 is a flow diagram that illustrates one embodiment of a method forreplacing texture in a video sequence and for reducing the bit rate ofthe encoded video sequence. Reducing the bit rate of a video sequencebegins by removing selected texture from a frame or from a series offrames included in the video sequence (102). Typically, texture from asingle frame is identified, while the identified texture from thatparticular frame is removed from a set of frames. This eliminates theneed to identify texture in each frame, which would increase thecomputational costs of texture replacement. After the original texturehas been removed from the selected frame, the removed texture isanalyzed (104) to obtain texture parameters. The texture parameters, incombination with a set of constraints, are used to generate synthesizedtexture (106). The synthesized texture is then inserted into theoriginal frames (108) from which the original texture was removed.

One advantage of texture replacement is that the synthesized texture canbe compressed more effectively than the original texture. The ability tomore effectively compress the synthesized texture results in a reducedbit rate for the encoded video sequence. The constraints are appliedduring the synthesis of the new texture to ensure that the synthesizedtexture is similar to the original texture. This is useful, for example,in order to retain the artistic representation of the original frames.However, the present invention does not require that the synthesizedtexture resemble the original texture.

FIGS. 2 and 3 more fully illustrate how texture is selected and removedfrom the original frames of a video sequence. FIG. 2 illustrates anexemplary frame 200. The frame 200 is a picture that, in this example,includes a person 202, a person 203 and a background. The frame 200includes several textures that can be selected for removal. Texturerefers to those portions or regions of a frame that have the same orsimilar characteristics and/or appearance. Usually, the background isthe primary source of texture, although the present invention is notlimited to background textures. One reason for selecting backgroundtextures is that these same textures or regions are present in multipleframes and in substantially the same locations. For example, if nomovement is present in a particular set of frames, the background willremain constant across the set of frames. However, the systems andmethods described herein can be applied even when a set of framesincludes moving objects.

In this example, the background includes several candidate textures thatcan be selected for removal. The background of the frame 200 includesthe ground texture 204, the sky texture 206 and the cloud texture 208.Each of these objects in the background are examples of textures. It islikely that the region of the frame 200 covered by the ground texture204, the sky texture 206, or the cloud texture 208, respectively, havesimilar appearances or characteristics.

When a region of texture is selected for removal, it is useful to selecta texture that has a spatial region of support that covers a reasonablepart of the frame area over a sufficiently large number of frames. Inone embodiment, a texture that covers an area that is larger than 30% ofthe frame area is typically selected, although the present invention canapply to any texture, regardless of the frame area occupied by theselected texture. In addition, selecting a texture that can be replacedin a large number of frames is advantageous because it has an impact onthe bit rate of the encoded video sequence.

It is also useful to select a texture that belongs to a class oftextures that are amenable to replacement. Generally, candidate texturesinclude regions of a frame that have similar characteristics and/orappearance. Exemplary textures include natural textures (foliage, grass,ground, water, sky, building facades, etc.), and the like that can beidentified in video sequences or test sequences such as movie sequences.Frames with replaceable texture typically have absent or slow globalmotion and/or only a few moving objects. Thus, the present invention canbe applied to particular frame sequences within a video sequence anddoes not have to be applied to the entire video sequence.

Texture Removal and Region Segmentation

FIG. 3 illustrates a flow diagram illustrating how a texture is selectedand removed from the frame illustrated in FIG. 2. First, a region ofinterest (ROI) 210 is selected within the texture to be replaced (302).In this example, the selected ROI 210 is within the ground texture 204.The ROI 210 typically includes more than one pixel. The ROI 210, forexample, may include a 7×7 array of pixels. After the ROI 210 isselected, the color characteristics of the ROI 210 are compared to thecolor characteristics (304) of all pixels within the frame 200. Thepixels that have similar characteristics are classified or identified asbelonging to the same region (306) as the ROI 210. The pixels identifiedin this manner are thus included in the texture that will be removedfrom the frame(s). The pixels that are in the identified region areremoved and are temporarily replaced with pixels that have an arbitraryvalue. In one embodiment, the replacement pixels have a constant valueequal to the mean color of the selected and identified region.

More particularly in one embodiment, region segmentation and textureremoval occurs in stages. For example, let a color frame be representedby a set of two-dimensional planes in the YUV color space. Each of theseimage planes is represented as a matrix and each matrix element includesa pixel value in row i and column j. More specifically, the Y framesconsist of all of the pixels {(i, j), with 1≦i≦M, 1≦j≦N}. First,original frames from the YUV color space are converted to the RGB colorspace. Second, the location {i_(r), j_(r)} is selected from aregion-of-interest, such as the ground texture 204 shown in FIG. 2, andthe ROI is constructed from: $\begin{matrix}{{{ROI}_{r} = \left\{ {{i = {i_{r} + k_{i}}},{j = {j_{r} + k_{j}}}} \right\}},{{{with} - \left\lbrack \frac{w_{r}}{2} \right\rbrack} \leq k_{i}},{k_{j} \leq \left\lbrack \frac{w_{r}}{2} \right\rbrack},} & (1)\end{matrix}$where the operator [ ] denotes “the integer part of” and w_(r) is odd.Alternatively, the region-of-interest of a size equal to w_(r)×w_(r)pixels can be manually selected from a region of interest. The pixelvalues are smoothed by applying an averaging filter to the ROI 210 ineach of the R, G and B color planes, and the mean vector [μ_(r) ^(R)μ_(r) ^(G) μ_(r) ^(B)] is computed, where μ_(r) ^(R), μ_(r) ^(G), andμ_(r) ^(B) stand for the mean values within the ROI in the R, G, and Bcolor planes, respectively.

Next, the angular map and the modulus map of the frame are computed asfollows: $\begin{matrix}{{{\theta\left( {i,j} \right)} = {1 - {\frac{2}{\pi}{\arccos\left( \frac{v_{({i,j})}v_{ref}}{{v_{({i,j})}}{v_{ref}}} \right)}}}},{and}} & (2) \\{{{\eta\left( {i,j} \right)} = {1 - \frac{{v_{({i,j})} - v_{ref}}}{\sqrt{3 \times 255^{2}}}}},} & (3)\end{matrix}$respectively, with respect to a reference color vector. The followingworks more fully describe angular and modulus maps and are herebyincorporated by reference: (1) Dimitris Androutsos, Kostas Plataniotis,and Anastasios N. Venetsanopoulos, “A novel vector-based approach tocolor image retrieval using a vector angular-based distance measure,”Computer Vision and Image Understanding, vol. 75, no. ½, pp. 46-58,July/August 1999; (2) Adriana Dumitras and Anastasios N.Venetsanopoulos, “Angular map-driven snakes with application to objectshape description in color images,” accepted for publication in IEEETransactions on Image Processing., 2001; and (3) Adriana Dumitras andAnastasios N. Venetsanopoulos, “Color image-based angular map-drivensnakes,” in Proceedings of IEEE International Conference on ImageProcessing, Thessaloniki, Greece, October, 2001. Notations v(_(i,j)) andv_(ref) stand for a color vector [R(i, j) G(i,j) B(i,j)] in the RGBcolor space, and the reference vector that is selected to be equal to[μ_(r) ^(R) μ_(r) ^(G) μ_(r) ^(B)] respectively. The notation θ standsfor the value of the angle given by (2) between the vector v(_(i,j)) andthe reference vector v_(ref). The notation η stands for the value of themodulus difference given by (3) between the vector v(_(i,j)) and thereference vector v_(ref), respectively.

In order to identify the pixels that have similar color characteristicsto those of the reference vector, the distance measure is computed byd ^(θη)(i,j)=exp [−θ(i,j)η(i,j)]  (4)and the mean distance given is computed by $\begin{matrix}{{{\mu\frac{d}{r}} = {ɛ\left\{ {d^{\theta\eta}\left( {i,j} \right)} \right\}}},} & (5)\end{matrix}$where notation ε stands for a mean operator over ROI_(r). All of thepixels within the frame that satisfy the constraint $\begin{matrix}{\left( {{d^{\theta\eta}\left( {i,j} \right)} - {\mu\frac{d}{r}}} \right)^{2}\left\langle {ɛc} \right.} & (6)\end{matrix}$are clustered into regions.

Next, the segmented regions that satisfy the constraint of (6) areidentified and labeled. In one embodiment, all regions with a normalizedarea A_(R) smaller than a threshold area, i.e.,$\frac{Ar}{MN} \leq {ɛ\quad A}$where M×N is the frame area, are discarded. The remaining regions arelabeled. If a segmentation map consisting of all of the segmentedregions is not considered acceptable by the user, another ROI locationis selected and processed as described above. The labeled regions areremoved from the frame and texture removal of the current frame iscomplete. The frames having the texture and color removed using thesegmentation map are obtained at the end of the texture removal stage.

To summarize, the region segmentation and texture removal proceduresdescribed above identify pixels in the original frame that have similarcharacteristics in terms of color to those of the pixel (location) orROI selected by the user. The color characteristics of the identifiedpixels or segmented regions are evaluated using an angular map and amodulus map of the color vectors in the RGB color space. Angular mapsidentify significant color changes within a picture or frame, whichtypically correspond to object boundaries, using snake models. Toidentify color changes or boundaries, simple gray-level edge detectionis applied to the angular maps. Identification of major change ofdirection in vector data and color variation computation by differencesbetween inside and outside contour points can also be applied. In thepresent invention, however, a classification of the pixels is performedin each frame using the angular map. Despite the fact that theperformance of the color-based region segmentation stage depends on theselection of the threshold values εC, εA and the size of the ROI, thevalues of these parameters may be maintained constant for various videosequences.

Texture Analysis

Analyzing the removed texture includes computing a set of textureparameters from the removed texture. In one embodiment, a parametricstatistical model is employed. This model, which employs an overcompletemultiscale wavelet representation, makes use of steerable pyramids forimage decomposition. Steerable pyramids as known in the art are morefully discussed in the following article, which is hereby incorporatedby reference: Javier Portilla and Eero P. Simoncelli, “A parametrictexture model based on joint statistics of complex waveletcoefficients,” International Journal of Computer Vision, vol. 40, no. 1,pp. 49-71, 2000. The statistical texture descriptors are based on pairsof wavelet coefficients at adjacent spatial locations, orientations andscales (in particular, the expected product of the raw coefficient pairsand the expected product of their magnitudes), pairs of coefficients atadjacent scales (the expected product of the fine scale coefficient atadjacent scale coefficient), marginal statistics and lowpasscoefficients at different scales.

First, a steerable pyramid decomposition is obtained by recursivelydecomposing the texture image into a set of oriented subbands andlowpass residual band. The block diagram of the transform is illustratedin FIG. 4, where the area enclosed by the dashed box 400 is insertedrecursively at the point 402. Initially, the input image is decomposedinto highpass and lowpass bands using the exemplary filters${L_{0}\left( {r,\theta} \right)} = {{\frac{L\left( {\frac{r}{2},\theta} \right)}{2}\quad{and}\quad{H_{0}\left( {r,\theta} \right)}} = {{H\left( {\frac{r}{2},\theta} \right)}.}}$The lowpass band is then decomposed into a lower frequency band and aset of oriented bands. The filters used in this transformation arepolar-separable in the Fourier domain and are given by:${L\left( {r,\theta} \right)} = \left\{ {{{\begin{matrix}{{2{\cos\left( {\frac{\pi}{2}{\log_{2}\left( \frac{4r}{\pi} \right)}} \right)}},} & {\frac{\pi}{4}\left\langle {r\left\langle \frac{\pi}{2} \right.} \right.} \\{2,} & {{r \leq \frac{\pi}{4}},{and}} \\{0,} & {r \geq \frac{\pi}{2}}\end{matrix}{B_{k}\left( {r,\theta} \right)}} = {{H(r)}{G_{k}(\theta)}}},{k\quad ɛ\left\lfloor {{0_{\xi}K} - 1} \right\rfloor}} \right.$where ${H(r)} = \left\{ {{\begin{matrix}{{\cos\left( {\frac{\pi}{2}{\log_{2}\left( \frac{2r}{\pi} \right)}} \right)},} & {\frac{\pi}{4}\left\langle {r\left\langle \frac{\pi}{2} \right.} \right.} \\{1,} & {{r \geq \frac{\pi}{2}},{and}} \\{0,} & {r \leq \frac{\pi}{4}}\end{matrix}{G_{k}(\theta)}} = \left\{ {\begin{matrix}{{2^{K - 1}{\frac{\left( {K - 1} \right)!}{\sqrt{{K\left\lbrack {2\left( {K - 1} \right)} \right\rbrack}!}}\left\lbrack {\cos\left( {\theta - \frac{\pi\quad K}{K}} \right)} \right\rbrack}^{K - 1}},} & {{{\theta - \frac{\pi\kappa}{K}}}\left\langle \frac{\pi}{2} \right.} \\{0,} & {otherwise}\end{matrix}.} \right.} \right.$

The notations r and θ stand for polar coordinates in the frequencydomain, K denotes the total number of orientation bands.

Statistical texture descriptors are computed using the imagedecomposition previously obtained. More specifically, marginalstatistics, correlations of the coefficients, correlations of thecoefficients' magnitudes and cross-scale phase's statistics arecomputed. In terms of marginal statistics (a) the skewness and kurtosisof the partially reconstructed lowpass images at each scale, (b) thevariance of the highpass band, (c) the mean variance, skewness andkurtosis and (d) the minimum and maximum values of the image pixels(range) are computed at each level of the pyramid. In terms ofcoefficient correlations, the autocorrelation of the lowpass imagescomputed at each level of the pyramid decomposition is computed. Interms of magnitude correlation, the correlation of the complex magnitudeof pairs of coefficients at adjacent positions, orientations and scalesis computed. More specifically, (e) the autocorrelation of magnitude ofeach subband, (f) the crosscorrelation of each subband magnitudes withthose of other orientations at the same scale, and (g) thecrosscorrelation of subband magnitudes with all orientations at acoarser scale are obtained. Finally, in terms of cross-scale statistics,the complex phase of the coarse-scale coefficients is doubled at allorientations and then the crosscorrelation between these coefficientsand the fine-scale coefficients is computed. Doubling of thecoarse-scale coefficients is motivated by the fact that the local phaseof the responses to local feature such as edges or lines changes at arate that is, for fine-scale coefficients, twice the rate of that of thecoefficients at a coarser scale.

Texture Synthesis

During texture synthesis, a synthesized texture is created that can becompressed more effectively than the original texture. In oneembodiment, the synthesized texture is similar to, yet distinguishablefrom, the original texture. Because the synthesized texture can becompressed more effectively, the bit rate of the compressed frames withsynthesized texture is lower than the bit rate of the compressed frameswith the original texture. Retaining the visual similarity ensures thatthe frames with synthesized texture convey the same artistic message asthat of the frames with the original texture.

A set of qualitative texture synthesis constraints are used to achievevisual similarity between the synthesized texture and the originaltexture. The texture parameters selected for texture synthesis arederived using the set of qualitative constraints.

In one embodiment, the synthesis of a new texture that is visuallysimilar to the original texture is subject to constraints in terms ofdominant texture orientation, and overall color and color saturation.Exemplary constraints include, but are not limited to:

(C1) Marginal statistics;

(C2) Coefficient correlations;

(C3) Coefficient magnitude correlations;

(C4) Cross-scale statistics;

(C5) Overall color; and

(C6) Color saturation.

If the original texture that has been removed from the original framesis structured and busy, the synthesized texture is subject to theconstraints C1, C2, C3, C4, C5 and C6. If the original texture isunstructured and smooth, the synthesized texture is subject to theconstraints C3, C4, C5 and C6.

Using these constraints, the new texture is synthesized by firstdecomposing an image containing Gaussian white noise using a complexsteerable pyramid. Next, a recursive coarse-to-fine procedure imposesthe statistical constraints on the lowpass and bandpass bands whilesimultaneously reconstructing a lowpass image. In one embodiment of thepresent invention, the order in which the constraints are applied is C3,C4, C2, C1, C5, and C6 for structured and busy textures, and C3, C4, C5,and C6 for unstructured and smooth textures.

The texture synthesis constraints are derived from the basic categories(vocabulary) and rules (grammar) used by humans when judging thesimilarity of color patterns. The following work, which more fullydescribes the vocabulary and grammar of color patterns is herebyincorporated by reference: Aleksandra Mojsilovic, Jelena Kovacevic,Jianying Hu, Robert J. Safranek, and S. Kicha Ganapathy, “Matching andretrieval based on the vocabulary and grammar of color patterns,” IEEETransactions on Image Processing, vol. 9, no. 1, pp. 38-54, January2000. In one embodiment, the basic pattern categories used by humans forsimilarity evaluation of color patterns are directionality, regularityand placement, complexity and heaviness and the basic color categoriesused by humans for the same task are overall color and color purity. Thefollowing table provides the meanings for these criteria: DerivedCriterion Similarity for constrained Criterion Expresses texturesynthesis Overall Perception of a single dominant color, Preserve thecolor or a multicolored image that creates overall color in theimpression of a dominant color the synthesized texture Direction-Perception of the dominant orientation Preserve the ality and in edgedistribution, or the dominant dominant orientation direction in therepetition of the orientation in structural element the synthesizedtexture Regularity perception of the placement (and small Not appliedand perturbation in placement), repetition placement and uniformity ofthe patterns Color purity Perception of the degree of Preserve the color(overall colorfulness in patterns saturation in the saturation)synthesized texture Pattern Perception of a general impression Preservethe type complexity based on the type of overall color of the overalland (light versus dark) or the overall color (light or heaviness chroma(saturated versus unsaturated) dark) or the spatial frequency in therepetition of the structural element or the color contrast

Because different visual pathways process patterns and colors in thehuman visual system, separate constraints for texture and color arederived in the texture synthesis process. Moreover, texture is typicallysynthesized with gray levels, i.e. luminance frames, and colorconstraints are imposed on the chrominance frames. In terms of texture,the synthesized texture should have similar dominant directionality asthat of the original texture. In terms of color, the synthesized textureshould have similar overall color and color saturation as those of theoriginal texture.

To address the texture requirement, the magnitude correlation constraint(C3) which represents the structure in images and the cross-scalestatistics constraint (C4) which allows distinguishing lines and edgesare selected and applied. While these texture constraints are sufficientfor unstructured and smooth textures, they do not allow synthesizingstructured and busy textures with appearances that are similar to thoseof the original ones. Therefore, for structured and busy textures, themarginal statistics constraint (C1) and the coefficient correlationconstraint (C2) are used. The correlation constraint characterizes theregularity of the texture as represented by periodic or globallyoriented structure in the set of constraints. The overall color (C5) andcolor saturation (C6) are the color constraints that address the colorrequirements. The overall color constraint may be further expressed interms of similar color moments (mean color and standard colordeviation). Because the pixel values in the segmented regions have beenreplaced with mean pixel values within the region, the overall color iseasily preserved. Because preserving only the mean color would yieldtextures with discolored appearances in the cases of textures thatexhibit large color variations, the color situations are also preserved.

By selecting different sets of constraints for unstructured and smoothtextures, and structured and busy textures, different characteristics ofthe background textures that are present in video sequences can beutilized. For example, by removing the marginal statistics constraint(C1), synthesized textures that differ in their first and second orderstatistics from the original textures can be obtained. Consequently,according to Julesz's conjecture, the synthesized and original textureswould then be distinguishable in pre-attentive (undetailed) evaluationof less than 50 milliseconds (ms). At a frame rate of 24 frames persecond, each frame is indeed viewed in a pre-attentive mode, i.e., forapproximately 40 (<50) ms. By removing the coefficient correlationconstraint (C2) for the same class of unstructured and smooth textures,the requirement that the synthesized and the original textures havesimilar periodicity is relaxed. This, in turn, improves the compressioneffectiveness.

The texture synthesized is mapped on the luminance frame of the videosequence. The segmented regions within the chrominance frames remainfilled with pixels having mean values as obtained by the regionsegmentation. Thus, the color constraints are satisfied.

Although the present invention has been described in terms of a videosequence, it is understood by one of skill in the art that the systemsand methods described herein also apply to a still image or to a singleframe. Thus removing texture from a set of frames includes the casewhere the set of frames is a single frame or a still image.

The present invention may be embodied in other specific forms withoutdeparting from its spirit or essential characteristics. The describedembodiments are to be considered in all respects only as illustrativeand not restrictive. The scope of the invention is, therefore, indicatedby the appended claims rather than by the foregoing description. Allchanges which come within the meaning and range of equivalency of theclaims are to be embraced within their scope.

1.-7. (canceled)
 8. A method for processing video data, the methodcomprising: removing texture from at least some frames of a set offrames in a video sequence; analyzing the removed texture to obtaintexture parameters; synthesizing new texture based on the textureparameters and at least one of the following qualitative constraints:coefficient magnitude correlations, cross scale statistics, coefficientcorrelations, marginal statistics, dominant texture orientation, overallcolor, and color saturation related to whether an original textureassociated with the set of frames is structure and busy or unstructuredand smooth, wherein the original texture is structured and busy, thensynthesizing the new texture is based on the following constraints:marginal statistics, coefficient correlations, coefficient magnitudecorrelations, cross-scale statistics, overall color and colorsaturation; and wherein if the original texture is instructed andsmooth, then synthesizing the new texture is based on at least theconstraints: coefficient magnitude correlations, cross-scale statistics,overall color and color saturation; and inserting the synthesized newtexture into the set of frames to enable display of the set of frames.9. The method of claim 8, wherein the synthesized new texture isvisually similar to the original texture.
 10. (canceled)
 11. (canceled)12. The method of claim 8, wherein the texture is synthesized accordingto whether the original texture is structured and busy or unstructuredand smooth. 13.-14. (canceled)
 15. The method of claim 8, wherein thesynthesized new texture is similar to but distinguishable from theremoved texture.
 16. The method of claim 8, wherein a bit rate of avideo sequence that includes the synthesized texture is less than a bitrate of an original video sequence.
 17. The method of claim 8, wherein aregion of interest is identified within texture of an initial frame. 18.The method of claim 17, wherein a mean vector is computed for the regionof interest.
 19. The method of claim 18, wherein the texture issynthesized and inserted further according to a method comprising:computing an angular map and a modulus map for an initial frame, whereinthe angular map and the modulus map represent color characteristics;comparing color characteristics of the region of interest with colorcharacteristics of all pixels in the initial frame; and including in thetexture to be removed pixels whose color characteristics are similar tothe color characteristics of the region of interest. 20.-31. (canceled)32. A tangible computer-readable medium storing instructions forcontrolling a computing device to process video data, the instructionscomprising: removing texture from at least some frames of a set offrames in a video sequence; analyzing the removed texture to obtaintexture parameters; synthesizing new texture based on the obtainedtexture parameters and at least one of the following qualitativeconstraints: coefficient magnitude correlations, cross scale statistics,coefficient correlations, marginal statistics, dominant textureorientation, overall color, and color saturation related to whether anoriginal texture associated with the set of frames is structure and busyor unstructured and smooth, wherein the original texture is structuredand busy, then synthesizing the new texture is based on the followingconstraints: marginal statistics, coefficient correlations, coefficientmagnitude correlations, cross-scale statistics, overall color and colorsaturation; and wherein if the original texture is instructed andsmooth, then synthesizing the new texture is based on at least theconstraints: coefficient magnitude correlations, cross-scale statistics,overall color and color saturation; and inserting the synthesized newtexture into the set of frames.
 33. The computer-readable medium ofclaim 32, wherein synthesized new texture is visually similar to theoriginal texture. 34.-38. (canceled)
 39. The computer-readable medium ofclaim 32, wherein synthesized new texture is similar to butdistinguishable from the removed texture.
 40. The computer-readablemedium of claim 32, wherein a bit rate of a video sequence that includesthe synthesized texture is less than a bit rate of an original videosequence.
 41. The computer-readable medium of claim 32, wherein a regionof interest is identified within texture of an initial frame.
 42. Thecomputer-readable medium of claim 41, wherein a mean vector is computedfor the region of interest.
 43. The computer-readable medium of claim42, wherein the texture is synthesized and inserted further according toa method comprising: computing an angular map and a map for an initialframe, wherein the angular map and the modulus map represent colorcharacteristics; comparing color characteristics of the region ofinterest with color characteristics of all pixels in the initial frame;and including in the texture to be removed pixels whose colorcharacteristics are similar to the color characteristics of the regionof interest.